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|^i||^out:aiBsr:Jthm tmgovts in full of three casaulttoica .3?hp, tackle^ 
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^^^^^^§^^b^0§^&c^3^^6^ff-^vi^ 1, Il» dnd'III type ERIC usage ,s«u^ 'designaj.;; , ^ 
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Data Oath^ring^ for the ManagCTentO:^ , ^ 
o£ .jE^G as ra^Inf ormation Natrori 



by Evelyn Daalsl 



ueedt Perhaps one reason lias In the fact thaf It Continual tp ptevide a 



stoiy of tha. six blind men with'-t^ alaphant Has been, greatly over-^ 



s 



pi 



drmatie and vivid analogy for a freqtientiy recurring phWomenon. As the 
coi^laxity and siza of our orgatiizatioflE Increases we hava baeopa accuatomad 
to people holding 4 wide spactrim of paremtions about the nature of the 



pirtiauj^r agency Mth which, they are daalltog* ;Thle is^^n^t a^y th^ lees 



vtwe when daalisg with ERIC, a systm (of spi^s) of wide varlaty and many 
contact ' pdints ~ a syst^ intended for great£y diver gant user groups with 
a ^rf4a range of purpoaasrsfot^ their usfe = i^s;systffl[X'Of diffusa and global* ' 
goals cto yie national lavel and graat diversity of specific objectives among - 
the various eomponent parts, " 

Obviously than* with subh a dagreje of differentiation in the ERIC systm, 
thtea must be a strong Impetus to^earch for intagratlve devices " ways to 




ancompaas and comprahand chis vast and ap^rawling young giant without ^inhibit- 
ing the flaKlbility an^ axperimentatlon necessary .for continued growth. Tha 
dynmic tension between the need for greateir integration and the 'opposing 
need for continued differentiation provides tlia backdrop for this paper, At^ 
the moment the decentrallzad charactar of tha ^ystm sam& the dominant fac'^ 
tor. Efforts Co imderstand and control, much less dlrecti tl^a growth of the 
systCT suffer from an ovarbalancing coward the dlf f arenciatlon^^d of the 
scale. « , ' 



i^^\-i.',u Lxn^^mxs paps 

S^ffY^ greater "^Rsordinatlom anH' cohesion for the ERIC ^scct- the 'first ^ 

1^:^ lliispggesrt^ is' f©r 'i^mbre hierarchical aeft?ofk artlq^atiom of the ERIC systan 



jfv-.J:^._\^e;'gecond suggestion ±^ £oi£ a 071 traa tic, /reporting syston as fart of a more 



\fQ3^ja^ mamagCTeat inf oraatlon system 



. W v Neither stiggestidfi is totally new* Various ap^oachea^/to deaqrihe the 
WXC systCT to the wbbv »ist* One recant ^m^lm.Mm the undated publication 
o£ deatlrai ERIC antttled. How to Use EMC » in whi^dh the* EMC systCT la da- 
soribed aa a "pyst^ ^^^th four levels"* (p, 5) xiie diagrw la provided beldw. 
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with four iavelar 
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:Thft^^ESt--^tK level ±1 tiepMsentad^by Cantrri, ERIC- 



. 



i-^-t ""(tl&elfet&dfet,, pollc^^^ ahd monitor) - : - The second or non- 



?f!SW^r?¥ri^^^^^^^^^ made-up of the 16 Clearinghouses located atr uni- .^i^i-^ " -m 

^--'^-^i^^^-^p^f^ commeraial 

r J^. ;i,^i^el ' ccmsfe cenfiralliadjifmclllties for msMging the .J.y 

and reproducing, doetmaaCSii.;\F^ the ua era who reeeire the ^ *:f/^^4;V 

" hanefit^jof tti#se actlvltleil* ^ . ' -^^^ 

Thfe f ojir revela presented are not .3eeally-ilerar'pMcat\a8 tli# word- "level*", 
miiht suggeat. Th^ diagria gems purposafiilly^^tjo have heen presested horlEOp'^^t 

.tally rather than vertically ta^prevTOt auch a mlsapntitptlon.; Further, I ven* 

* ^ ^ " -. ' ■ . ^ ^ ^ ' ■■ 

- ~ ■ . V 

. it i #. = ^ ' . ^ , 

ture to suggeat,. this Heserlption falli to provldeia uaefal fraaewo.rk f of j^om* 
preheDdliig the MIC aystm either from the Vlewpolat of the uaer, the at^ndljig'^^ 
order euatomert the elearinghouie ataff^ and perhaps even from Central^ ERIC 
itaelf / ^The vtewpoJat Brnmom llbrary-lika with hea^^airphasis on materiala, 
their aequisititfn and processing, ratTier than servicea for the uaer* 

A somewfai t different ayatCTl articulation/of tW ERIC net^rbrk will be 
presrti|Cad this paper as a more helpful f rmework for cdmprehending the 
aystCTi from^ a variety ofr, peri^ctives that' of the f under, the Manager ^ the 
^taff j and the user* 

The second suggestion derives from the flrat. Given a satisfactory de-- 




=^ription o 



f the a^tCT, aystmatic t 



iiclmiques for data gathering; about use 



in a managTOsnt information system frmework can be devisad. The many use 



atudiea by Md for 



have provided gross figures on uae* uses* and users j 



but they lack co^ia^ability and they lack/lnharent directions for ^ahag^ent. 
A good managCTent information systm will provide. data for decision Mking at 
all levels of the ERIC systm. A tentative .description an^d method of develop- 
Ing an appropriate managmcnt Information' systm will be presented. 



\ • 



f^'^-^^ln^yS^ig' pabM-, - tfiSi"; "di^^Sl^iato three parts or ^ ^se Isf " examina t ioa;:^" 



'^'^i^iending'J tem^ng, altd mansgtag the ERIQ eystra.,. ?a!i^t B tfjj entitled > Management - ^ - - 



'Sy8t^ , describes tisrful dat^ (jategorle?.* api^^dpriate levaJLe df '"I(^ 
X ^"data'eollictCea, aid suggfests] a 'tf^tlAd: of ae^elbping'a'mgdml datd eqlleettag'"^ 
_ ^^^knim* Part C\ - ^rij^tled "S^efeimehdatlang i ^atairnk^ thf a€^KBu|gtot$QM^ 
. ) of jehe^art' of the^ioeeiKLe*. r ' • * r^ '-- ! ' - ' 

''f;^' / ^ . " ; ^ ' . * ' ■ ^ ■ . ' * ' i ■ ' '^^ ^" 

First* it Baay btf hiiipful to diactiia "^loae prior deeie ions that weta nefi^s-- ^ 
f4ty betqie dfeveloplng/an ^aitarnat^e frMework, Thi '^J^? cQneern TOS^whetHbr 
nfe ahptjid .fa lionceived /a ^urvayOr of produots pi^whather ERIC 'a pr:Uflary 
^bn was to^^pMvide seryice^/ A aecond ceifcern. focuae^ on the standing . ■ 
ordain eua&sM-: 'Ware thara advantages ^to seelnj them as mCTbars' Of tha MlCy 
aystCT rather than euscomari only ' who atood Outalde the boundaries the IRIC 
sypte^^ A third prdblttS camttfred ott tha olearlngh&us^s with their dual fu^e.^ 
t£on"* What should he 'the ptlma^^ alleglance^ofs the elaarlnghouaa ~ to .the 
toational interest comttnity,* o£ ta the local geographic comunityfs Given that 
bdtji ate In^ort^nt, what rjiqri4titat±on# should take place and/or whkt con- 



fltcW mlgfet* foreaean and preveoted? The fourth ^oblm cone emecr the hier- 
archyri If thare was valCye to conceiving a single hierarchy for the ERIC systOT,, 
%at were the approprlatfe lavels and what ware the unique reaponslbillties of 
each of thMe lavals? Finally, how were the systm elements related in the 
network? What dagrae of indepandance/dep^^ence sMould each elOTent have? 

SystCT of Products VS. S^stm off Se rylces. Current manag^^ent thaofy has ^ 
becooa incrakgingly rasulcs-orlanted rathar than actlvlt^rlented . (Por^a 
good ovarvlew of current argmnents on this topic, see Waldhart, Thomas J., 
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f^^s^^^t±v±ty-^oT±^md vies?' o£ fianaganerit, MnagCTent was sien a syst^ 
l -iS!^^^^^ ^hasizing euelv ddtivltles organizing; dl:reafcingv control- 

V % as a r^i^rc© t© the^ organization and raeults.^re atreaied though partlclp^" 
tlob and coordlriatlon* Tha straas on reaulta foeuaea mote .attantlc?ni>pn put- ^ 
comas and lass on Internal actlvltlas. Activity-orltete^ manag^^nt mighty ax- 
praas objaitivea In tirma of products even though tha products ^re only liastru- 
' mantai in aehlaying the ^ctual overall goal of the organization* 

If BMC ia saen as "a systCT of prdducti p then a ralativaly simple market- 

i^^ing^^del, J.ooklng -.at .sa eosts^i^aM^^tjsti^ns , ^ gMgi^^ 

\ . ^' ' ' ' ^ ' ' ■ ' -. 

^4 ntmber of continu4ng customers, etc.^^ would be aufflelanti It is apparTOt 



from the OTphaala placed on the end use that this product orientation la not. |^ 
atcaptabi^/ Rather a view of ERIC aa a syatw of iarf^isea aeens the appro- 
prlata^model and haa the. added advantage of allgOTieiit with current rastilta- 
orltmted mahaggaent , theory, ^ ^ ^ ' = 

2, The Standing Order Cuatomer . Many of the user studlea focua-on^ the stand- 

Ing* order customers, (A particularly useful one Is that prepared by Eleanor V, 

Homej A Profile of ERIC Microflcha Collection Sites , ERIC Clear Inghousa on 

Tests j MeaaurCTent, and Evaluallon, Princeton, NJv:1976.) Theae SOCs can be 

sara In two dlfferarit ways. If ERIC la conceivgd pf^as a aystCT of products, . 

tha SOCs are. the retall^s and the end polntv However, looking at^ERIC as a 

syatam of ^ervicea* the SOCs become the prtoary outlet polat, the place whe^e 

the end user Interfaces with the ERIC syaEra* This means that the SOCs might 

helpfully be considered; as elraynts of the ERIC -ays tm and not external to it* ^ 




ifeaS^^l itfett^is 'of policy ^nd -allocat±ort-of "readurces , -it- woul'd-mean- greater ^-f^-^ 

i^Sliiiii^|iE,th^.SqC^ :develoVifl®nt., of .Incentive plans , to_enepurage trie SOCi^,-^^^_^^^^^_^.;^^;; 

i^ ^' cp ralSp atod a etrongar hlMarehlcal network Mnp^t% '\ 

^i^rc ^-^f ||ee||rita^ vmTf In the amount ©f dont^et and coticqrn with ' , 

i^N'MlharlodafMd the^ do not understand or -recognise the GlearinghouAe, role. 

^f0^f Gee^air Sbc#has great ^pli^atlon for the 'ClearlngKouse^- / : 

^ CO which we ttowitura* " V ^ . 

3;' The Dual ■^uttqtlon of the Clearinghouses , In this coneeptlon o£ a more In-- 
tsgrated, \cotitrollable systCT, the Clearinghouses aisume central Iroportance. 
/it la they who will be the middle managers t coordinating policy* plans, and 
resource allocation for Central ERIC to those SOCa in their geographic purview, 



and at the same ttoe* iyatenatically collectittt feedback from the SOCs to pass 
on to Central ERIC* 

The Clsarittghouses presently vary widely in their CTphaaea, and no doubt 
' ; rightly, ao- Mostj however* focus strongly on their national intereat comun^ 
ity and spare little, time mnd attention for the^SOCa in their geographic region. 
There are many inherent conflicts, of course, in asking the Clearinghouses, to 
asaime the dual function of suparvlsor/facilitator to the JOCs in addltlbn to 
their prescribed duties of produce acquisition and development and marketing/ 
consultinE contacts with associations and specialists in *their subject field, 

t 

Injthe system coordination aspect of the Clearinghouse, there would be a 
^ number ^^htdditional general education and corfs^tation functions required by 
• the Clearinghouses^ as they ''big-br Aier" the SOCs, but ^rfObably the primary i 

additional role would be as collector of reliable use d^ta in-^C systCTatlc 

" " ^ ^ . ( 

fashion. To some extent, the ClearlnghQUses presently^ perfom the fomer gen* 

eral education function. They tMght spend more time in referring eiM u*sers^to 

■ • " ■ '--v J 



^ffi^frappEOpriat^SpG-^^^ direct user , Interface for. occasions 



H % '' "" ^liie egtabiistaent o£ 'tKe ^ClMrlnghouse as the primary eeilecting point 1^:1 'tl^f^^^ 
/-^'^ v^.f ot ;uie data* wooLd liavm a nmiber of advantages, 'Presently,' Central ERIC, 



--'^^cettalaalons national user studies to docimant the value' and ^thfe Impaet of ^^^A^^^^vii 



^^^^NQ BfBtmL' ThB data aolledted, however * are often grpss' eat^iates and 
iOiffefa unreliable. Perhaps even some Inaceuraeles acctit-^n the* present use 
data colleoted by the Clearinghouses ^ dt leart to 'the extent that the d|Lta 
does not" sdem inherently useful to thM: aiid is only bjeing colleotjid for re- 
porting purpoaaa. The accuracy of the data would undoubtedly be-.enh If 
It ware meaningful and useful to the level where It is being collected. 



. ' ^ * A Hierarchical SystOTS li^ewSrk ^ ' ^ - p 
'figure 1 on the following page presents a view of. ^ the IRIC system whlah 
may be\a useful and ttoely wa^to conceg^|uall2e it. Each of the thre^ levels . 
shows has its ^ own distinct responsibility and requirei c^M kinds b^JLn^ 
forEBatlqn in order to fulfill its responsibility competently. The Information, 
neadad by each level will be, in most cases* suomarie| of the information , 
neadad at the more articulated level. There may be.valua in extolning this 
systm from the bottom up, giving primacy to thfe grass roots levet where most 
actual use ^curs . 

% r ' * 

SOCa, is shoTO in Figure 1, are categorip,ed into one of three types, 
^pes 1 and 2 are quite different in their institutional constraints and in 
their intended user groups. Type 1 SOCs are the college and university sub- 
scribers* They are different in that they are constrained 'by the university 
enviroTOent to serve those users who are mesbars of the university, A second , 
topoitant aspect Is that they s^e ERIC aa only one of a nOTber of educational 
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resources available to thm. Ug^ rlar^. hoipFMi f-r f-m^ gt-ntjp, must: 

™-rirefiefct both the eonstraint^ ^nd thm l^t-o^T- ^^i- ^6 ^^mijs^U]^ roctr,ure^s, 

Typ€ 2 SOCs are thosg^that sarv# pviHHr Qrhocl p^r^onnel . These are often 
. sponsored by stata education agencies. Some are statewide computerized net- 
works such as :^SIS in Umj York Stace, SJffiRC In California, TRDN in Illinois. 
Others are based lu a i.^|iX.,.. - .4 - - i-^cs- ^i*. 1,.. i 

I ,».... 1 i . i , 

. . 1 
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on population to hm served, percent of pnpul sMon ^H^^ -^r© ctirr^nf users, pmr- 
Cent "that' are rmpmmt ummrB^' p^r^mnt th^e si-a f^'T-«f- us«»rs, ate* SOCa also 

naad Inforaation to help thais pr€^ct whan h«?flvy use cttn^s ulll occur. Tht^. 
can be pradlcted in part from data on types of use relative to month of year. 
Central ERIC may be leii tac.reiCid ii : i cite-bpuad aspect of this data. 
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To perfoifm the second mission^ acting as a rp^glonal node in ^ n^r^onal network 
lor distributing. lEIC pyoducts and serylcei, some of the same types of actiyl= 
ties are nacessary — the linkages, a pxihlf ration program, direct provision of 
Inforaationj instruction, but hers thay are directed to a specific geographical 
regional- unll: aiul cin,! : » ^ n( \: \ ^ ' \ '~ > i ' i" in.. 4' Ifl^H 

systm and sarvxcau . j^in u , imi i.i . Mk4a. . 

dl4^^ti^ i.. ^.»J - jKfl . . . , J 



squrees for pre^detarmlned InformflriDn n^^^s . 

Many. daJlnitlQas of a m^tm^mkmnt informael^^n b^b^'^^^ ar<» avallabie. Most 

J 

saan to include these four iaportant* alemanti: / ^ 

CI) The WIS is used as a basis for decision-making. ^ 

% . ■ \ 

Ic is a s£ 



respect to the sllacarlon ^nA tlt ^i^s^^r»n r^sourc^^, ^ *- ^1«3o needed 

dgnoQSU&^cs os^aoiMtioaa-l f activeneBs aad sffi^^€"f"y ia ths use of re- 
sources. In addition, as th&ra ar^ no she^nlu^^ ^g^Quroo oP parf^rmancA for 
the SOCe. thm Cleartoghouses , or Central FRTC ro As9e?5R perf orm^nre against, 
laformatlon axchangii ciCi | t^ldi ( c i| i i , ' < ^it 'lie bails of compariscr 



lip CO record aeriial uja*. t-^joe , -»**'^ m^*-^" ^ ^ ... ' ..f ^ p^* 

formed on the data at intervals by qWli^^^'' ^^r^-^-^ ^^'^ a-^^ -oi i pr^r r>M cvc / 
teia should be k^pr 3^ qlmpi^ ^^^^a^h^» 

Third, data should be collected on a sampling basis. It ^ uc«i;?11v rhe 

case t hat grea tti 1 r e c c S 1 * ^ |= = = ^ == = ^ ==. ==== 



tha, latter i-Qceotlv© is a powerful one. 

3, Tentativ e HQdel . of a Data Collactlpn Syslam . AprWing fh^p r^^esant data 

/" ' — - ' _— ____ 

eatagorims usad by the Clearinghousss to a simple matrlK* Figure 2 dCTonstrates 
how/'th€S# prografflaatlc functions can be mapped into a series of prograia meaff- 
ur^ . Financial data is cgbi 1 it:. ersasMcIc tui^iioLm 
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Thla Inf ormatloti simply fleshas out th^ r^««^rin report fng ^f^^, prouiding a 
^^jtui^mr piatujSB^ of activity and resource use. It should prove helpful In jn- 
craaslog control and. in dayeloping tisc»^iO longifudlnal <?t:vidles of Clearlnlhouae 
acftLvlty, , 

deV ^1 up i, * ^ , i » . I i < t . J . J 

k y 1^ £ J £ 3 ^kA^ V , ^ . i| . i . 
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C. RecQmettdations 

If the suggeatlons in this paper seem to hm ^orthwhiie goals ^ what would b 
a reasonabla plan of implementation? For org^n f ^«ff on as large and as di- 
verife as ERIC a totally new systeta would be ey'c^eedlngly difficult to imp lament 
The suggestlDnr hnrn prrrrrrc' rf^^T"^ r^^rf ^hn r^^ailabla data^ collection 
ay^Lgsi and e»*iv,fiii i > i * ..^ . j » ^ .1 si s . ...^ i a/> 1 l / i^- ^i^aiui 1 1 
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(8) Istmbllsh staodjr4;^€i^ri]^oLogy ^and defimltlemB for aecagjCfille 
data eatef dries # CstisldM cstei©rlas devaloped im«lfiaB 



Hatlosal: Standatds Institute. 



(9) Drvelpp etaadsE^d forEst for uniform ree&rd kseplng td^tti optleoal 
/ " eategoiflsi that ean Is deleted and opportuQitiae to add cate- 
gorite for otKM loaal required Inforaatlen. 
^ (10) Uaa a^art to heLp astablish appropriate a^^le iiges aftd' fra^ 
quaaaiM -for reaurrlBg uaaga data* * . 

(11) Aaslit ClearlatbQusM in developing a rolling data colltction 
plam for all SOCs In their geographic regdsn so that all SOCi 
ara iOTplad at leask onca every five years* 

(12) Saleet a set of SOCs that are alraady keeping good records and 
that include a range of types, sizes j and gaegraphlcal areas 
and set up a ten-^year longitudinal study. Soma Incentives 
may have to be provided to th% parent Instltutlonsr 

(13) At periodic intervalSi engage an expert to analyie the descrip- 
tive data collected la mora sophisticated ways* 

(14) Provide periodic data gtirroarles to Cleariiighouaei so that they 
eao compare their rtsults with the aggregate or wltfi selected 
others like thOT on certain characteristics. 

(15) Rislat temptation co collect «ore data in greater detail at any 
particular level than i^rn necessary for matiagment decision^ 
making . 
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(8) Istmbllsh staodjr4;^€i^ri]^oLogy ^and defimltlemB for aecagjCfille 
data eatef dries # CstisldM cstei©rlas devaloped im«lfiaB 



Hatlosal: Standatds Institute. 



(9) Drvelpp etaadsE^d forEst for uniform ree&rd kseplng td^tti optleoal 
/ " eategoiflsi that ean Is deleted and opportuQitiae to add cate- 
gorite for otKM loaal required Inforaatlen. 
^ (10) Uaa a^art to heLp astablish appropriate a^^le iiges aftd' fra^ 
quaaaiM -for reaurrlBg uaaga data* * . 

(11) Aaslit ClearlatbQusM in developing a rolling data colltction 
plam for all SOCs In their geographic regdsn so that all SOCi 
ara iOTplad at leask onca every five years* 

(12) Saleet a set of SOCs that are alraady keeping good records and 
that include a range of types, sizes j and gaegraphlcal areas 
and set up a ten-^year longitudinal study. Soma Incentives 
may have to be provided to th% parent Instltutlonsr 

(13) At periodic intervalSi engage an expert to analyie the descrip- 
tive data collected la mora sophisticated ways* 

(14) Provide periodic data gtirroarles to Cleariiighouaei so that they 
eao compare their rtsults with the aggregate or wltfi selected 
others like thOT on certain characteristics. 

(15) Rislat temptation co collect «ore data in greater detail at any 
particular level than i^rn necessary for matiagment decision^ 
making . 
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(8) Istmbllsh staodjr4;^€i^ri]^oLogy ^and defimltlemB for aecagjCfille 
data eatef dries # CstisldM cstei©rlas devaloped im«lfiaB 



Hatlosal: Standatds Institute. 



(9) Drvelpp etaadsE^d forEst for uniform ree&rd kseplng td^tti optleoal 
/ " eategoiflsi that ean Is deleted and opportuQitiae to add cate- 
gorite for otKM loaal required Inforaatlen. 
^ (10) Uaa a^art to heLp astablish appropriate a^^le iiges aftd' fra^ 
quaaaiM -for reaurrlBg uaaga data* * . 

(11) Aaslit ClearlatbQusM in developing a rolling data colltction 
plam for all SOCs In their geographic regdsn so that all SOCi 
ara iOTplad at leask onca every five years* 

(12) Saleet a set of SOCs that are alraady keeping good records and 
that include a range of types, sizes j and gaegraphlcal areas 
and set up a ten-^year longitudinal study. Soma Incentives 
may have to be provided to th% parent Instltutlonsr 

(13) At periodic intervalSi engage an expert to analyie the descrip- 
tive data collected la mora sophisticated ways* 

(14) Provide periodic data gtirroarles to Cleariiighouaei so that they 
eao compare their rtsults with the aggregate or wltfi selected 
others like thOT on certain characteristics. 

(15) Rislat temptation co collect «ore data in greater detail at any 
particular level than i^rn necessary for matiagment decision^ 
making . 
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^^^|ii«r2Srootfti6aiMri?^ cmptufed. this, w^, , , '\.^r.[^^-rL 




^\i^MS^^^^]p£!^ES^_^^ MBm_ patterns arid longer term changes and developmenti aan be 
S^ W5€ttdiaid walli^ THereforej thia paper will deal with aoa-experlmental ^ approaches ' ^-^hA 
to ERIC use EsaeeCTenta. nie goal is to obtain flm esclmatea of use, whllB 
avoldliig the eKpens^ of^esnatis* This Job Involyas a c^rafulf aequaatlal e^e^ 
cytlbn of spealflc tasks » twe af 'whlsh deee^e special attention here-, 

F^rst, souad amp ling, decisions ansuce the' representativeness of Che 
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\ group which will provide li^omatlomp Steond, only valid; questions rfnlve 
valid aaswers. Getting valid answers from representatlva respondents Is what ^ 
Is needed. Getting valid answers from unrepresantaclve respondants defeats 

rganarallsabUltyr how broadly ^our results apply -r ema las imelear* ^ If^ the 
respondlmg group la repreeentMclve» but the answeri ^re Invalid, the quality 
of meaiiurraant has baen diluted; whether your reaulta and conclusion c^n ba 
trusted baoomea quastlonabla. To avoid suah pitfalls, sampling must be baaed 
on a population which la clearly defined and accessible « Also^ the Information 
requested must be within the scope of what respondents are capable and willing 
to offer « 

The complexities of the sprawling* decentrallzad ERIC syatffli have clearly 
presented a challenge to reeearchers who try to study It* The difficulties * 
could be at least partially alleviated by narrower conceptualisation of the 
studies' goals and priorities* Fry (1972) recognised that in his final report 
by saylagi "The esctent of use . . by the total universe of users - . * cannot 
ba estimated with . * * the present study," (pp. 1^11) Actuallys we may say, 
chances are that no one study could assess the use of the total universe of 
uaers^'-^aa this is too global a conceptuAliiatlon under present ERIC systm con^ 



f^^^^^ons. J tuira^ shDttceadniS 4a the pait hi;vfe eapcgedc 

J^J^^Jlil the areas of ^ s respoMe ratta, amd aeasuifcaetit. - : . ' ' ^\ 

Sr^ ' ^1: ; , ^TAs^^H^ out Elreadj, miTvmf studies off IRIC ainas' ' 

;Ja., =the.syattm!s laaept have typically tisad purposive or tise aampleis not par-* 
Mtttng %meMliiafcloft, Furlhermotei if aueh atudier attCTptsd evaluatlost 
y thay wuld hav€ apo to people already predispoaad to 1MC| II they ^nted 
CO meaaure us€p thal^ estlmatea were recdertd unreliable « Apart £rra the 
saiqilt's ripreEentatlveaaas t there la also the deflnltloii of the pool from 
which reapoadents are to be chosen* One half of tha aurvey sttidles deals with 



eduoators regardless of their oomneetlon to EMC* Thus, the mph^fis of study 
la laadvertsntly placed on intended, rather than actual audiences, and Mount 
ot ERIC use may be severtly undereattoated* Once again s (1972) warae of 
the coBse^uOTOes by aaylng that i!^the field la mo vast Md diverse that ^aly 
gvoss est^iates of ERIC usea can be extrapolated to an umeisured total unl^ 
verae of educators.** (pp, l-ll) Non-EMC connected gMrplta are good enough If 
the goal Is to estimate the proportion of uiars In aarrowly aelected groups of . 
potential audieates, as done by Denby (1974), However/ if the main thniet is 
oa actual ERIC use, then sampling must be done of the "catetoiat" areas where 
contact with uaers can be measured. The identified mass of us^rs could then 
be broken dovo into more specific categoriea of relevance. After uae and users 
have been depeadably determlnad, work on non--ustrs will be easier. Reasons for 
non^^ae, discontinuing ^ rejection of EBIC, eqnld be traced , and ways to enhance 

i 

the role of ERIC mong potiatial audiences could be deviied, 

Uasatlsfaetory response rates plague even the. bast of survey attmpts 
(e.gi. Home, 1976), so the rellablllcy of Cheir reaults cannot be determined. 
As Havelock notes, studies where mailed quagtionnalres result in response rates 
below 70% are bound to generate dubious data. Yet, such 1m response rates come 



j^au.^&,.aurprlse,. glveavtliat^ nmtled guaatlOTOiAyBg have been long fcaown is the;. 

i le^t^fultf ul apt^foaQh Co data collection. It la odd that motm rese^chara 
hava nok tried L to aahamas th^lr tespomse rates eomhlnlng.maillag momm 

,cf . the toswa ffietholis, oft Tha ^flation In data QalLeotldn 

approaches bBiAm been'^min^U, ^th mailed ^'queatlonnalyes In the do^nut eate- 
goryi as seen in. studlsa reported &y Havalock (1977): - \ 

Method ^ # of Scudleg ^ r ^ ^ 



mailed queatlonnaire 27 

hand delivered questionnaire 5 

_int©irviews i , . . / . ; t: 

personal ' ' ^1 

group . 1 

phone ^1 

abaervation 1 

sxperikent '2 

recordi ' 2 



' ' The seri^uenasa of measurCTent problms Is tied to the above -meat toned 
pethodo leg leal choices, but also to difficulties stermnlng from a statt of 
at fairs at ERIC system, ©utlata. For eKanrple, in reviewing data, eollactlon 
methods, Havelock singled but tw studies which braved data colleetlon through 
records j with this coament' ''If forts to collect such data are generally re- 
ported to be hoplass because of total lack of standardization in record keep** 
ittf on users/' (p, 16) Both Fry and Home' therefore^ found themselves in the 
difficult po alt ion of drawing cone lus ions based on uneven quality data " part 
^oming from firm records, part beaed on respondfemt "eatlMtes", The word 
"istifflates*' means guesses offered by rispond^tSi In the ahienct of ^#yatraatlc 
records or pbeervatioris * The ineluslcin of Impraaalonistlc daM can defeat the 
best of, well -designed atudles. .Without belaboring thepolnt , it la important 
to undprscore the vast dlffetence between survey estimates baaed on; f Im data, 
in contrast to data which represent "estlniates . , * by library and inforaatlon 





KSiiiiWi®^|||J|S^ me fbrmar aliaws the calcU^ ^ 

kJ=r^§<-.i^S5i3^«?.4t^K^^^^ ..- „ - . =. > • : -J : . ..v*-'■^^r.-•=■ 



ftfciMiSg^att'eras' aiQdng Relets tuggaata that a problra does'^inaefe^v^^dLtt 



|3s:|(Sae "Appendix I.)- .' =\^j^.r-'^ ' ^^^^^^^^ 

rrr ' tor axa^lSj msng SOCa, school iOCs tand to kaep good records oh uatrs 
..^rjand macsrlalai uttlvarsl^ library* iOCa keep aoa^ rfeeorda oa^Mterial circulsr 
l^^ tlon aad leiMputer iemfchts, irhtle BOGIS SOCa t^^lcally do^not keep raebrda, r 
" TMs immediately suggeaea tbat the sost pro^sing SOC tutgrbi^ whare use aa^ 



%:S^ssiaamt could be aoeOTpiiatied veuld he ifiM cuCTent 
inforamtlon, ira are told that about 72% qC^^ES^C SQCb ata placed alsawhara 
at eollega and tmiverslty librairieat where raeord kaeplsg la apot^* So 
, -jr^^iag sffort on the small gToup of school SOGs alone- would aontrihuta 
' llttla to our toowlfedia of overall SOC use pattarne^ In the Bamm valn^^onie ■ a 
atudy found that 63% (m**160) Tbf Che 254 SOCa who regpondad to her atudf 
iome raaords* - Since we dq sot know what kind of sa^f-ialection led to. these 
' particular 254 SOCa respoadlag, out of the totality o£ 50Ca contacted ^ wa-can- 
mat faiow what the probable record keeping proportloh for all SOCa fflght^ ba# 
The study also alludes to a further cosipllcafcion of the 160 re^eord keeping 
SOCSt only 91 ware willing to share their information and records! that ia, 
36% of all responding SOCsj or 56% of the record keeping ones* and an unknown 
\ proportion of alJL elating SOCs* . 

By contrast we see evidence of ease and detail in chf ©SIS study by 
Wllkaa (1976) s whan records ©f computer search requests were available, the 




time period, useri and unit of analyala were clearly defined. 

% r Given the abdve, the iBmedlate concern of current sasessnierit studies raerges 
as Bialnly one: resolving the problCT of availability of uniforTn use data (records) 
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^lil^addltioni facorde ihould become asmewbat mora eomi^slla from pl&Qm^^o 



^ place. ,A£ ,tlie same time, it ®ust be. reeogni^ed that a drive f ©r such account- 
mBility presents a delicate problem in the larger plcttire of library operation 
lEKCCis ottB of mmy tools available to patrons i as itich, its role^n a epical 

librlry is not fneeeeimrlly c^tr^, from the point of ^iew of library ataf f 

" . ^ - " =^ ^ 

and Information speeialiats. Tbus» there ^y be practical 1^1 ta to staff 

^ i. * ^ 

willing^esB to devote ataable mounts of t^e workings on MXC records. 

^ t 

On ^ broad plane, ERIC use assessments can benefit moit from a progrm^ 
maticj Incremental approach, where consecutive studies add on to the ovetall 
ilia ' picteitf maimer . A tagtilarVJ Wng-tam so^tdrt mhd"^" 
evaluation (laspact) program should be drafted and gradually laplCTented by/ 
Central ERIC, if at all possible* Such studlea could have the following . 
^asirable attributes v . , ^ ^ 

(1) ' Clear frame of reference , narrQwer cdaaepturi definitions * ' 

of user and use , with corresponding operatlonall nation and 
measurement • The priority should Initially fall on actual 
* use, rather than potential or desired use, and user popula- * 

tions. Smaller, valid studlea ,build a trustworthy picture 
faster; global, multi-purpose studlas generate needs for 
. further validations, 
, . (2) Clear separation and reporting of results based on fina 
records from th^e extrapolations basad on unsystematic 
observations, o't opinion* 

(3) Random probability sampling of clearly listed "populatl 

(4) Choice of data collection method, or combiaation of meth- 
ods, which enhance response rates. 





*£qi£ WIC Use Assmmsmmtm 



^5SKJ^^:ff:£^^ ... .. . . . . , ... 

^M3^#H^->4^Let^^ra^ Co sanm q£ the concrete stape that can te tafcea toward 



^o«e;,?=^|MMseffient use. Our foeus will be >a dcrfflestlc 30Gs and ME sub- . 

gfg^SjlciSLB^M statistic^ suigest the folla^ng ^nto dtag^mf 




3^427 by addreas fUa fcat 
2,938 by Paultoa's laap 



= n f 584 by 1977 miC Dlreatory 



n f 42 by Pauline -a map 




A* Au^efltatloti of Directory of ERIC ?llcroficha Collectloni* 

Additional ittformation collected v^ia ERIC 'a queitlonnalre for the directory 
*ls seen here aa thfe first buildini block toward valid use assaaafflent study de- ' 
signs « Currently i the following nine ^tms of iriforaation are being requested i 



J 

na^e 

address 

phone 

contact person 
I collection status 



equipment i a , microfiche readers 
b. reader/printeri 

services I a, hard copy reprpduetion 
b, microfiche reproduction* 

chargea 

access days /hours 



We propoae augmentation of this questionnaire by three more it^B"^- Inloma^ 
elons " ' 
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add c* cQttputer baSe^'seaichirigi and 



i^S^S^S^Sgfr^:;- "-r, -"d.-Tendliig oE -Mcroflclie^ readers andrmhe^Ur^ T^^S^ffit^ 



^^^^JzJ^^d^a -second paga^ior a new:;lt^ headlag - ^-TecQi^is^qt^m^^A^^ -^fm^^M 
fKy ;^ 4iGl^aiys^ aupimhtatipn, the quest lanoalre will csnsist of* two pages. 



Directory of ffilC Mteroflche ^Ueetioaa 



1* instltutlen namm 
2. addreM ^ 
3 * telephone 



4, 
6, 



eontaat person 

eolleetlon status (ymtB eovared) 

equipment available, ' 

a. # of mlerefleha readen 

b* # of mlcroflehe reader /prlaterg 



aervlets available 

a. hard copy reproductlom 

b. oierofiche reproduction 

charges 



p yes □ no 
p yeri . □ ao 



e, eomputer base searches p yas no 

d, lending microfiche readers p yea □ ao 

e, lending flche □ 7M □ no 

8* days/hours available to the ^public 

Page 2 9, records on use (see below) » 

\ * 

10. conmeara 



1 



As a result, we now ^111 have infonnat ion useful both to Directory readers 
and for the purpose of designing assessments of systm use. More apeclf Icaily , 
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We. propose the following Iteais for inclusion under records on use , page it 



records on use . 

Please indliaata whleh of the following usage statlatiGS you 
keeps if anyi 



it fiche circulated 


□ yes 


□ 


no 


# flche duplicated 


^ n yes 




no 


t paper copies" "made 
# manual searches* 


py^s 
□ 


□ 


no 

^no ^ 


f computer searches 


□ 


□ 


no 


it orders from EDRS 


□ yes 


□ 


no 


^usw of RlEp CUE, Thesaurus 


□ y®^ 


P 


no 


i of clients served 


p yes 


□ 


no 


types of clients served 


□ yea 


P 


no 


purpose of clients 


□ yes 


P 


no 



other usage* records (pleast escplain) 
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^^^^^^^^/J^^Q^osSt' inquiry bti reeords given €bove is purpoiyslyr^brief ? m the 
^i^?#^4naitf?ioal at this stage is to prepare an accurace list of ^^the SQG diehotomy — 
KSffi°i=^^t±ose with -a^ without usage staclatics/ Once completed > th£s phase o£ 

in f btaatiori gatfiaring will i^rva as tha basis for "the SOC iurvay whicl^ wa ari-^ = 
t ^Slonj datallad data,^ll he requested at the time of survay^^plMaaCatioti%. 
; \ - The disauaslo,m of such a survey appaars in later pages of this report* * 



B. lUl Subscriber Questionnaire, 

The ne^ct atep would be to turn our attention to the RIE aubscribera. At 
the present time^ we have a listing of these subscribers, but we lack much In* 



formation as to their collection, equipment, or iervlces relating to ,ISIC. - In 
a nijpber of cases we ecpect these to be pinlmal. It would'tie desirable, to 
approach RIE subscribers in a mamer similar to microfiche collectlona C^OCs)* 
Periodic eoinpilation of a comparable RII ijibacrlber directory could be the ra^ 
suit of such an effort* 

To thla end, all taiown SOGs will be. eliminated from the list of all known 
domestic Rll subscribers. Using Pauline -s map datai this would mean that we 
shall eliminate the 591 knOTO SOCs from the 2^938 taioTO domestic Rll aubiarlbers, 
thus arriving at 2 5 347 eligible RIE subacribers* 
V^enever a known RIE address Is listed for more 
than one subscription^ It will still be counted 
only once, ^ Using the Venn diagrm again, we 
are now Iboklng at the shaded population r 




1. Normally J we can aasimie that all SOCs subscribe to RIE. However, rtsearchers 
should be alerted to possible anomaltes. For example, according to 1976-77 
data* Arkansas had 3 RIE iubscrlbera and 8 SOCs, which leaves 5 SOCa without 
an RIE subscription. The appropriate checks should be made for thtfa and 
_ similar anaomallea at the time of any future study* 
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, . . .. .... . . . . . _ :s;5i&i^d^tmh- ^y^^' 

^^^y^^1^G^&k:dnB'^69%..wmwi^ ,€stlmaeas, an ME diraet©iry^wuld|^an four tJ^es % * 

"Maalmapy quM QO^raipondlngly large publicatiaa^fecMapated^fb the -J ^t^^S 



^?$&^%ixm^xyrj^ N^eirthaltsSi tht purptose of 7#ontactl^^E aubicrlbefs; is; ^to ? 
i^l^'^tribliBW a baasiima of Imf ormation unavailafile previously; thuS^' a cotfplate / 
:''4dysrvlw (Genaua),, rather than samplings is racoramanded as aarving tfie goal of 
i: full d#scripti©n best. The magnitude the effort ean be liffiitad by reducing 

the scope of the questionsmir a submitted to each RIl subscribiiV* Thus, wa prp- 

. * ^ - ' ' '' 

poea the fdllowini itms for InetQpion on one pagei 



Directory of IlIE^Subsaribers^ 
Questionnaire 



1-. 
2* 
3. 
4. 

5. 
6. 




8. 



Inatitutlon name 
Addreas 
Telephone 
Contact person 

RIl subscription acatus (year started) 

Other ERIC tools available: 

CIJE □ yes □ no 
> 

Thesaurus p yes □ no 

Equipment available' 

Microficha readers p yes p^n^ 

Readers /printers p yes □ no 

Any ERIC related services (please describe) 



These should be printed in ^h^ad of ttoe, , 
BB they ara^taiown* , The reipondant would Tiave 
the opportunity to mark any aorreetioni or 
changes that might be neceiiary. 



□ no, 



charges i 



9, Any ERIC related usage statistics you keep (please dMcribe) 



no 



10. ComentSi 



erJc 



THANK YOU YOUR TOffi AND COOPERATION! 
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purposely opeg^^nded". ^ r^"^^^^i^'C-- = -- ^ : 
data prep'S^K^^Sulated^ 'a more ' - 
eSe ^ For examjple , 



jpSfilM&^Ml^descr of a6n*S0G IIB subaarlbers 

I^SHIE sjiWd^'ibera be :4i^d^ =^latQ reXevaac groups, 1, e.#,bT; availability ^ of ^ 

^ toals^^ other than RIE, or aqulpaan^ and asr^ces related to ^^C^ Host Impor^ 
^ ^ tmntly, a^^^ based ©n^Ertaifaoe, or abienae oi^&ior^B ^9)^ nm^ be 

gatie^^^r Agaia, the Vrnm dd^agrup sarvlng ^aa pur^ wrklmg frame^ of jeef ereaces 
ss^^lftak^ atlll one step further i .■ ^ , * 

ME keepijag usage racards 
R^E net kei 




eplag reoords 
— SOCa keeping usage records 



^/ — SOCs nof' keeping rtcords 



Data Base Searching 



Soflis surprialng results Jaay Inttoduce changes In this schfflie. For eKample, some 

f 

noA-^SOC Ell sub^cribera may report access to data base starching. 



C. Surveys on Usage, 

At this * point a projeot diractor may be ready to begin actual ERIC usage 
attzdiei* design* Thia is Bo because, assuaing that previously described stepi 

V . * ' 

fiavs been taken and the designated' information obtained^ two important tasks 
have been completed i 

(a) Clear separtion of ERIC outlets keeping records from those 
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not keeptiig records Is new possible. This means that 
further down the Itns, rsporting of results based on firm 

sltuacloas. 
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places the recdrd keeplrig and non-record ke^^plng SOCs cbrotighout the USA. Then, 
tvo separata random samples are to tffe- drawn from either population* Thus^ the 

SOC f would :.u <>i >. >i < \ > ^ \ !\>^i{^\iJL iliap lciil)t:uLed In 1.1 

Little i li D\i X>vai - - -i ' - i * ^ . . uu. 



tli% SOC dlrect(?ry atid gr encoded on tha form sent to respective InstitutiOTis . 
provision for corr^cclng or updating of pre-coded Information should be made. 



Era J ,11 ; : I 1 . ^ I 1 J . 1 



* ' I.C.J 



I 



How would you characterlzt the p\^^rm^r^r of FPir mflfc»rial<5 and 
services at your iHStituclon : 



consolidated in ont location 



p 1 1 
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# eoomufllty Individuals 
9 go'^inmeiit (local, state) 
othari (specify) 



agency staff 



12. Purpojiip q£ cll t h 
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S'Dllsg^ 9 urns NyrcK 
chases i»y4 dla Gkla. . 

d s V i. 1 



i I 4 
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17* tl Caacral ERIC prepared such a comon SOC record-keeping f6rs* 
wou4.d ymi be tatarssQed infusing- it? * 



n no 



The aott-ffg-eocd k 



* v^O ^ . , 



,C 
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survey to be implCTentfed as a "mailed qusstlonnal re only" study, given the 
dismal response record Vl^^t avlous attempts. 



4, , 
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Level III ERIC Usage SCudy 



TTl-7 

In any amplrical ssrtirW FPTr ■(flips<-f' f-fn f^^ct, in ^^11 ernplrlc-al studies 
In all of the sciences) th<a i nves t i g;ir or Has tvo ultimACe concerns Ot questions: 
(1) to what cixtiiTit n il ' '"^ ' ' ^ ^ cti ' i v i 

aa4 (2) wU^i cx t 4 i « , . i ^ , . ^ 
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' ettd^uaer would require Che following four ^t^ps 

1* Randomly divida tha currtnt and potential end-users Into two 

for Che dur-atlcii t < . ; * ^ i i ^ <n «t)* ^lil s^!^ i- 1 Osi^ 
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* 

To aoalyze this as a true field <=ypf»TiTnpTir , an airprnat"Jve q€»t of fout^|r*' 
steps need to be followed, 

1, If all of the i Ci ace .1 t i v. v^d lu th^ study, here 

are eUw-ugh uf « ih CO ^ l ^ . ^ i ^ v ...... 44. i t l l Uat Ic , 

1 A s, I 1 _ i 1 , , > ... 1 * . - * . * ... - i * » 

i» » M ^ i i i i . 4 . . • » - - * 
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ITT-5 

(e) Never discard a unit from the list because It 4oern*t 
natch very veil with Its paired' unit. If one or mctm 
units aeed to AiannvA^d , do tht^ f irst (hntox^ 
st€p a^) aad tl P' nilbLi, t-iatl^ia.i/ >ose whicli anitji 
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TTI-6 ^ 

1. True experiments requira that some units (end-users) bm 
denied the proposad chsHie. Since the change would not ba 
recoinnended unless It wab believed to be better than the 
aurreoc ^sl ^uat i an, i i ^ ai ti e : l u i . i i . : 
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i^Witfi regards t0 ERIC iri is doubtful that dmial of tha pro- 
"bosed* change In poliGy or procedure will be as harmful as ^ 
aay ths denial of a new medical treatment to seriously illv ' 
patients; Horeover, the denial of the proposed ehanga is not 
permanent ~ it need only eontlnue for the duration of the 
study, W^th little additional effort , corrective dctlon can 
be taken after the study has determined whethar or^not the 
proposed change is. In fact, .att ^prov^entl TOo^ end-users 
iti the "poorer" group can be given thSSpportumity to make 
use of what was available or permitted in the "better'- group. 
In faett if the impact study leads to^ an ijnprovement in ERIC 
then those in the "poorer" group may profit from the study by 
being able to avail thenselves of the changed systm. 
Wifle It la easy to concei^ of the iituatlons in which true 



experiments cannot be carried put as in the measur^ent of 
sptte ongoing* unchangad operations — true field eKperijaents 
are possible more often than might be believed^ 

ERIC impact studies deal with the effect of a service or 
product given to an eod^-user. If a product or service can be 
given, it can-alsp be withheld. So at _ least potentially * one 
of the requisites needed for randomization exists in mostt if 
not all, impact studies* 

This Is important to mphasize because the atep-by-step 
experimental ^guldalinas above have dealt with only one way to 
effect a change in the end-user, by changing ERIC. This is 
not a requirement for an eicper tmental Impact study* The re- 



quirement is to effect a ehahge, in the life of the end-user, 
_and as mentioned ahove, this may be.^ccomplished by with- ^ 
holding services instead of by changing ERIC. _ * 

Thera are other Indleations (see also Cook & GfflnpbellX 

.--^ -■ 4l--->- - ^ iii 

whleh suggest that randomization may be pbiaible, ' Wienever 
^he nimber of reqvests for a apeplf Ic EMC product eKcaeds 
the ability of the systCT to eupply that productt the ahoice 
among requeatora can be made rmdomlyii. Ors in more^genK^al 



tarmi * whenevar a proposed^hange la dlf f teSlt * "^Tanaive j 
or time^conetmingi it cannot usually b^^plenented all' at 
onc^ eve^?^here in the system, A random implraentatlon plan 
win allow trua field e^erimants* ^ 

One potential difficulty with field ^cperlmenta la con- 
tamination between those who are to receive the proposed 
change and tbcs^who will not* This problm is alleviated 
somewhat whenever the axperJAantal units (and-uaers, CHSs or 
SOCs) are unfaiown to each other or ara aaparated apatially 
or COTp^rally* Since this describes many ERIC end^users 
(and some CHa and SOCs as well) s true field axperlmental im- 
pact studies of ^RIC are an even more attractive alternative. 



Many true field expartaents ha^^ conducted in medicine 

(which may seem to be more ^e^ble to laboratory-like pro^ 
cedures) , in criminal justice (e.g., who should be raquired 
to pay bail) I in education, in welfare refona, in television 
advertising* and in other areas. It Is difficult to imagine 
that they would be impossible to conduct in other areas* such 
as ^n ERIC impact studies. 

Some manufacturing plants use a quality control proce= 



■^Pi * fclvemeaa of the product (a ee Box and Draper). 

In addltlra to these t^o general obJeccionSs top^et studies of information 
"J;-- a^stras ayggest another* more specific objection, 

^3T"^"Ti^ field S^pW^l^^ are so/ rich witff^ufi¥dn^^ 

^and the meaaurea available to assess impact £tm so weakr that 
any real difference in impact between the two graupa of end- 
. j uaars will not show up in the resuJ^s, The true results will 

' be either hidden in the noise of the other stimuli ^ or not 

' \^ maaaured with instrument a sanaitlve enough to discern thCT. 

Without denying the premlsas in this obj action, it should be noted that, 

3^* tteny non-experimantal studies of ERIC have cpntalned some 
'assessment of Impacts These studies usad impact maasuras 
which dif f erentiitad among classes of people (such as users 
and non--usars) . Because the only key ddff erance between 
itjiese atudles and true field experiments is how people be- 
come members of tha user or non-user group » there la every 
reason to believe these s^a maasurea can be used in expari- 
mental studies of Impact* 

B. Other Rlaearch Designs. 

True field experiments ihould be the first chQiGa of a design for ERIC 
impact studies* And as the previous section arguad, such designs are possi 
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dnfe"^ the^feffecfc of a change in policy or procedure. Ugage.^attidies in infoCTa- 
ion ays terns are^eapeolally vulnermblei the Tsanagto eoffipar W th^ previous set .- 
f ua^is fipires with ctirrsnt data anS attributea the difference to soma j 
pecific change whiah ocourrld in the inters. This duign should not he used« 

Deaign XX is muoh batter. By t^king^iiaa of atf^eral before and after 
leaaurest a careful riaearcher is more to identify the ^effect Qf the 

hange on the end-uaers* It ia important, that the different measurpients be 
he sane ■ or co^arable — the same instrisaenta and observations taken f rom ^ ■ 
dmpara'b IV samples of end^users * lit her the same 'group of end-users""can* h^e"^^* 
easured evtry time (as In^a panel study) , or different random staples 'of end^ 
aers'can be employed. Each alternative has its attractive features and its 
iaMlitiea, Ihe panel, study is subject^o syatmatic "dropping out" of par- 
ieipants, biasing the panel and possibly causing wjiatevar differences are ob- 
ained between the before and after 'measures. Use of different random samples 
llminates this problem , but prevents assessmant of the long run effect 
Impact) on individual end^user%. Analyzing Eha results of this design Is not 
traightf ©rvard and will require some expertise in statlatlcai techhiques (see 
lass, Willson, and Gottman) . 

Design III needs to be described in more d/itall* It requires two groups 
f end-^users' the members of one group are able to avail thOTselves of the 
iiange, those In the other group are not allowed to do so* This design differs 
a three important ways from the first true experimental design described 
irlier: (1) Design III requires one set of before measures from both groups 
f end^usera; the exparimental design did not use any bfifore measures. (2) 
^stgn III San use Intact collect tons of end-users — such as two SOCs ; the 
cperimental design requires char rh^ groups be created through randomization.* 



rhe alfiernatlve true experimental design presented earlier makes use of Intact 
mics also* The f und^Tf^onf al ^jf^'^^^n^^ ty^g ^^pp^^^^,^^^ Design ITT 
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0Ufh^p0y uses a coin-flip to decide which Iroup Is to" be given th€ change 

f^^P-r' procedure or policy; the experimental design umms a coin-f lip_ to. d^^^^e . 
i-^^t,^ ^ ^ . rr f^"p tngTTii^ g|i ^ P for each CTd--user, - ^ - . . - 

^S^I^^jJjS^ ispi^ettted in the^iSlowiiig mMjW^^'* toici^the^BUtfpoaea 



||-::i^i^'^f s^^lifyiittg the pEeBratation, only two units Cnaturally occurri^ collec- 
::f ' tions end-^uaera) will be cgnaidered. 

'\. ' ~- '' ' 

: ^ .^^V Identify the two imits. Ideally, these ahonld be aa stoilar ^ 
. - ' : _.aa poaaible ia t«M of^ ali!.measures related^ to!. tapact, J*n4 ^ 



- ^therain lies most of the difficulties with this design. If* 

: \- ^ ^ ^= " ' ^ - _ _ - v 

* ^ the two units are selected becauae of availability or by ad* 
miniatrative ruling * they may not be almilar fenough, if' the 
researcher is permitted to choose among aeveral avmilable . 
. units those two which are most aimilar, the selection proce^ 
dure itself may decrease the trustworthiness of the findings'. 
^ To avoldj but not prevents the latter problem, the before 

measures should not be used to find similar units, norj^should 
any end-users be ^Jfecarded from a unit in order to improve 
the degree of simllaricy. 
2* Make or permit the proposed ehange available to one unit and 
withhold it from the other unit* The choice between units 
should be deteralned randomly if possible. 
Devise an instrument to measure the Ijapact of the proposed 
change. Try to have all @nd=-userSt or a large random sample 
of all end users from each unit complete the inatr\ment. 



has to 3o with Che conclusions. The alcernatlve true txperlmental design must 
havennany units and the results «pply to units- Desi^ III only needs two 
units (it may have more) and Its results apply f-o the end-users. 




i^^i-r ^^'^^^^^^ ^^^^^ two' u^ta were sliillar^ enough on all esstatial-^varl- 

. 'Sblee» th^ any difference in tapact between the un±tS|can be %i 

^^^^ - / ^ . ^ 

%$it^^- . - - attributed to the -change. = - 

f Desl^ IV eomblnea the attributes, of De^gna O and therfe are two 

^ ' ^nlta of &id-i^ers md each la meaaured a^eral tlAM before and .after the ^ 
r change is impl^gnted. As before^ Irt la linportant that the t^ units be . ' 

f^^: sliailar .and it would]! be helpful if the choica beft?een the MnJXB (which *ia ^ 

to receive the change) be determined randomly. ® "fl ^4 

'\ ' C, Hsaaurement Conaideraliona* \, 

^ These include who i^%eaaure, how to maasuri t^fi, and what to^ mpaawe* =^^ ^ 

; These are general recommendations ".specific impact measures are diacuased 
later In the paper, * ^ ^ 

" ■ ^ ^ ' 

1* Obtain mteroisaasuras as well as t^c^tmeasurag . King and BryfnC distinguish 
' " Ve^|eaa these as" follows* ^cromeasures are aimed at determ\nltfg the total im- 
pact of the intormation sjatCT or a component of that syst^* On the av^age, 
how well is d.t functioning? How affective Is the inforaatlpn syatas? 

Questions of turn^aroimd times accuracy of information, tijaelineas of in^ 
formationi coverage of collection, form of output, and cost ^e some comiott 

macromeasures . They are at the macro-level baeause an* aggregate conclusion is .1 

- ^ = / • 

made aboUt .the average response or the ganaral tendency, , ^ *r * 



Many investigations of information systms deal only wi^T^PM^tiiromeaw 
possibly because they are easier and lass costly tq obtain. But the Improviment 
of any operation depends upon the Incrmental elimination of failures (instances 

of poor perforaanca) . Mlcromeasures are a study of Individual falluras. Why 

-- - 

«^ * 

did this particular unit have such poor performatoce? Why was curn-around time 



.r:>so-lorig3 Why *waa it more costly thm .othmrs, etc.? By examining each' instance - 
of failure it la possible to determirie if the aituatioa was atypical and ought. 

.^to ha ignored y or if it can be traced back to the sMe lource as other failures 

^-^^ ^ ^ ' ' ^ = ■ ^ 

. — In^^tich case rmedial action needs to be^takM* * , 

When impact la being studied the're are Several major pQtentlM- sources of 
frilure arid it will often be dif ficul% to reliably determine Jthe source of thoee 
failures* " * . ' 

coilectiori failures^ the selection or production of ERIC prod-^ 
ucts, the feoverage of the data basis the accuracy and recency 
/ of materialgi etc* \ S , " . 

^ organization failure ; the arrangement and elasslf icatioji of 
* products .rfj 

„ delivery failures^ the'^ availability of products and services 
and the timeliness of their delivery to the end-user 



2, Measure non- user 3 as well as users . There are ti#o conderns here. First 
there i^"^ th^ question of frame of reflTAncr; If IRIC Is the focal point * the 
Investigation Is of users ruS their role, reactions ^ behaviors, ecc. as a com^ 
ponent of ERIC, This frame of references while valid and often useful , is 
limited* It cannot deal with educators who obrain ERIC materials indirectly; 
it precludes a microanalysis of non^users " are th^ ^'failures" of promotional 
^terlals* of accessibility and location of the SOCs, or of pricing structurejj - 
and it eliminates any direct comparison of ERIC users with non-=users — which * 
if systematic differences were found, would certralnly jffect an^T^ conclusions 
drawn about the Impact of ERIC. 

The alternative p^r^p^rfive. rhnvigh mnr^ costly to Implamentp Is recomflended. 
It has people a.^ th^ fnr^i p^inf^ ^n-^ ^. rr>\p and Impact of IRIC 1^ 




> ■ =,=4^^^ is aia" aid ooet but well worth itepeating evea tiiough- 

^^'S problM, To ^atudy urtrs one must d%al with 

^ r#^1 iig^i^' 5^^r^g^^^ y:a)prs'*atich aa e^ar-ts^or judges* Surrogate uMra 



can judge^^j^act^ xiaefulnesSs quality, br yelkvanee in a ^ery limited sanse, 

■ \ ^ ' ' 

Moreover J iurro^ta u^ara have a difficult time inti^aetirig with ^IC whieh 
^ also llffiiti tBe uaefiilaeaa and applicability of the obtained data (aee D* Swanaon) 

And finales it should be p^ointed out that a comparlion between us#ra ani 
r^yS^^^^^^i^ i : cannot adequatelT^er^A^ES a^u^ 

'^ ^p arisen tal d^lgn. True e^erlmenta TCqUire "identical" groups for eo^arlaon. 



Since people sefec^ thasiselves into a uaer or non-user category, theae gro\ipa 
are rarely identi&l. ^ 

3, Qbtain_^yatwortt^ measurjanents^^ * To obtain reliable and valid measures 
careful preparation, training, and pretesting are essential* Using several 
measurea rather than will help* as will unobtrusive maasurment. 

The vast majority of ERIC, impact studies make use of quaati^naires or 
interviews* .-Ihe specific wording, JorMt, and order of , each question of itm 
can potentially affect the response or the investigator Vs Interpretation of 
the reaponse (see Hoelle-N^mann) . , 

One common* example of invalid measurOTent in the study of information sys- 
tems occurs when the Investigator asks potential users their needs, but gets 
instead their expectations. Frequently used technical terms are often ambigu- 
ous. to respondents lowering both the reliability and validity. Of particular 
concern are terms like "usefulness", "satisfaction", and eapecially "relevant". 

Researchers should also be wary of using the general global opinions of 
end-users as a major impact measure . 1 ^Carefully measured opinions can provide 
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'hifB^S^j^mJmmLtoJiQYaTall funeclsaing" of the systCTi, and eaeh end*-user's' 
^ieiponsm^caojbev^ analyzed ms a iniGromeasuraTA But a major weateese ©f sueh 
5 ©pjjilons is tliat mo relatloMnlp with the = quality or jiapact of 
the system — partleipants * resposdTOts, and end-users ate likely to be quite 
fmvorahly disposed toward'^lRIC even if a more objective approach identifies 
serious ^waakaesseg (see C* Weiss, p- 41)* The safest^ way to iiitei^ret the- 
opinions of end-'users is by cOTpar^^ than ^th a suitable eomparlson C^ugh as 
is an eatpefijaental i^tudy)V ^ . ' • . ^ 

^^i^^EUC^iiapaet studies^ott need to Jpeuf the end^^wjfegjrtettMtm 
past I quest iomiairas and: toterviewa ask about ^ prior usagi ani the imjpaot of 
tiitt usages General questions of this sort can provide trustworthy infomation 
only if the end--user's mCTory is accurate. That is not usually a safe assump^ 
.tion^ To miniM?e this problam, it is helpful to focus the e^d^user'B atten- 
tion to a specific',; recent interaction with ERIC and 'frme all questions^ in ^ 
terms of that "critical incident". . - - . K ^^Wr^ 

While single measures whlcl:^are highly ^ip^iable and valid are .desiraBlei 
they are often hayd to find. Another apprgach .(which should be used ip con- 
junction with the Improvraenc of each measure) if ttp e severri related meas- 
ures and look for some comonsllty In response. Several measurea which agree 
are clearly more trustworthy than a single measure. Though it helps to ask 
the same impact question of ERIC users in several different Waysi it is still 
better to base an investigation on beha^iotal im well as verbal responses. « 

i x:, 

— Did potential users report that ERIG has as* impact dn their 
professional lives (verbal)? , 

Did potential users dCTonstrate that ERIC had an Impact on 
their profesaional llv^s (bahavioral) by uaing ERIC p|oducts 




^^^^^^^W^HcrV^^^^ia^'^mR*^ cltiatioM in mrtlcles written ty the- , = 



'praetices in the classroemp etc*)f ;' 



some of ' the-tahavioral rMponsee Is chat they may - 
tte'^bhseri/ed tuibbtrtslvely ^ without the user knowing they are being Qbaswed* 



« ^.If the process .of aeaaurraent doaa not Inpihge the uaer^it eMnot Retort 
; Jtodings, Cheeking citations fir ^ Sources provided\by ERIC is an ejEafflplii :/^ 

©f unobtrtisiva measur^enti^ . , 

While there are hundreds of iaeasurea pcoppsed or used is th^^^- assessment of the ' 
^erformmca of an information systm* there are -^e:^ few coimohly used measures 
of impact. It is clemr'Trom the literature that the state-of-the-art of i^act 
measurement is still in its infancy. Part of the difficulty has to do with the 
subjective, non-quantitative nature of^^^^act^esessment . And part of it has 
to do with the fact that practicalxv all of the other measures used in system 
assessment are IrTelavant (or almost bo) to ^Impaet. For exmplej in a study bf 
^ the performance of an information- systenj ^c^uracy measures based on relevance 
(e.g., recall and preci&lon) will usually matter. However i in an impact study 
of the same system, end-users may find the retrieved Infoimatioa to be -useful^ 
^ helpful^ and having an effect on .their prof essional^ lives regardless of the 

^ ^ V ' ^^^^ ^ ^ ^ =^ - f 

relevance of the retr^ved dociOTents Cn t4ttoa, of t original information 'needs. 
^ ^ In measuring impact, it is important to Wep'in nftnd thaC ERlC end-users 
are like eve ly one else — they are not always systematic in their searching be- 
haviors, they often change what they want and what it is wanted for, and they 
are quita willing to follow an unexpected inforaation path if it looks interesting 



. am..lndebted to Professors Ittchael ^HcGill and Judith ^Tessler for their assist^ 
ance in the conceptual development of this saetlon. 



Er|c ' bT. 



if' 



?ffi%t!^'kSaaB^!^tieH;bahavloral indicei meed to be aubstantlatad with some aaaeismeAtB^/rl, 
^^^^jgf 4hov? ^h^cltmtlgna helped the eduaator. ^ 

Aftef revlewlftg^iphat little lltaraturs there la on Impact , three somewhat 
' dverlappiiii^meiiurfea' sMnd^^^ aavlffgd * Qj^^^y* cMslder- 

" Ing each ©£ tl^ie iadii^4dually , two ''r miaders mmm appropriate, ^Flrat of all, \- 
£^>^.^HtIfi:^faets:;nte 
5^.-^^Si^4ta official outlets^ (e.g. , SOCa, CHs , EDRS) , hut" also Jthof a ^Am^tB jm% ^^^rM^ 
tain ERIC materials or benefits from EMC services in^iraetly (e*g* * from other 
edueatora) , Impatt studies should lnclude>tfth types of end^users. Secondly : 



tlje intact of an information syste^may be quite a bit more than the sum of the 
^Ji^&cta^f each ^formation itm, IRIC uSers may interact with an intermadlary 
imA refer to the*" th^aurus before ob.cmlnlng d octants. Each of - these contact 
can affect the user gome meaningful ways and a thqrough^^pact study should 
include the and^-usarls total Interaction with ERIC, ^^i^. 



A. Savings. 

One tyi^e of impact Is evidant if an ERIC product or sarvlce can be shown 
to aave tim# or resources. It is difficult to Qbtslti any objective meas- 

ure of such savings. Coaper*s "naive methodology", though difficult to imple- 
ment In practice, la in prlncipla on the right track* He suggasts that users 
be asked to estimate the value of each docifflient produced by the aystem and 
quantify that value by assigning an "taount in dollars (which may 'be positive,, 
negative, ot zero). The user judgas each doc^ent to the order In which they ^ 
appear from the systM; If the user changes his mind about the topic, if he 
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= - ^Dollar estlinates for each documa&e can be summed to obtain a meaaure o£^y 
thm utility of the total s^areh Interaction. By conrparinj these values wtth 
those obtiain^a frdm alternative Versions' M the i^otrotfon systmt Cooper 
hopes to obtdln a useful meaaure of.syst^ miimatl^mnmBB^^ - ' 
^ The -'naive methodology" can be extended to ^ aaaess Impact* . Haerl can be 
^aakedi^po^astiJttatei^heH^^ savlngs-whlch mlgtit^^iee^ult from 

laformatlpn In edch doetOTent* Or they could aatlmate produaptvlcy* changes or . . 



tlse savings^ and trai[Mform these into dollar Mounts* * * 



^1 Vh» 



Such subjectiva judgments need to be interpteted carefully* One apprdach 
which wlll. help is the use of a comparison group™ by coopering the subjective 
dollar ^pact judgments between the ^oups^ biased estimates of the difference 
In iBpaet betwaen ERIC and *ah alt«rnativa source. "It would also help if each 
uBBt is dfked to focus on a specific education problOT or project and describe 
/"^l^nj so^r^dftaill where the savings ara to come from, ; / 

^=%'^^^^^Sa^tngff , in time can ba assessed in a similar manner* However, a true ^ ^ 

' ■ . ' _ . 

field, esparlmeiit with several "aft;^er" measures will provide a more objective* 
' -measure of^^cr^e aspect of time savings' not savings on the Job, but the amount 
of time ERIC saves in locating inforpation products. By comparing EMC users 
with noA^users periodically over several months , one could learn how long it 
tak^g the non-users to "catch up" with users in teTjas of their knowledge of, 

r 

' familiarity with, and use pf various information sources and products, 

- Some discussiotis of impact measures propose Chat avoiding duplicating pre-- 

vious research is a t^e of fime savings. Though this wlli* stfve the user time, 

i 

^ It has questionably ImpaeC on th# pr^jj^saicjin -as a whal#. Edueac-ioo,^ like many^. 



TTT ^1 

/ • 

of the appllad social sciences, n«^^H<= more respar-h rep J i t i on <=; , not fpver, 
if ic is CO be able to separate Uioj^e EcstiaiLh its^ults which ar^ .r: 
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Easa-of-use and rssponse r-lme arp two p«sr f ormance criteria which provide an 
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Approach for Measuring Impact. 
Impacts on prof ission are ralatlvely easy to measure, but a study of 
this eflect of: KHIc; uu iM: m < . i,*=y....s , <s (cuni ot pub 
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solved or they may have already been resolved* Wliat matters 
is that the respondent can recall the problani In some detail, 
hi 1 Import ant ^^^/^^^l^ ^Jt ths pruUi^**. ^ic ti.cii iiou^d is .-ju uii 



0. 



ERIC 



and related fields. Question (d) is the '*how" question for 
personal impar*^ ^ -^^r" nn (a) ^^n-^i/ ^f- — *nsur' 

a failure amlysli . ^ 



TTT -'11 



A, Basaarch Dasigti and General Haasur*>m«»nr rnn<! i H a r 1 nn« . 
I c a L 1 f I i t i ' ' ' i\ 

flLt.l4itly *i i.^- 

L . A i . t ft 



— **What «€re its bad or awkward features?" (Timbip Coomhs) 



In chose atudlss ^hl^h ia^ludfed udera .iJv . 
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g respondents to estimate thg effect rhat rhplr use of FRTC has on their stu- 
li §. Sl«ber used ." t ■/i>;Li:j ^ 1 j:, . 

, .r.f. 

li ■ 1 . . 
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time savings) which supported Che user's reply* 

% 

— Savliigs w€re implied >^ the gual^^v^f i < I : ..i j ! t. it" erred by iil&s 
or .subscription flguL Sl. (Burchliial/ * 

' L L ^ . ,t L k i, ^ k i k titbit t»^t,»».4 i 

^h^. m) h-»ve y^u uLl^4 o<id ^ hem *' u . ^ . • . . ^ . . . i .i 




^ion lmportan| materials that you probably wouH not see e 
where?" (A, often; 3, occas'ionally ; C. naver) (Fry). 




" If you aeafl Inlarmatloii lq auiva an ^ini^^jjciiiL pcu^lam 1 
work, what Is the farthest that you wQuld be willing to 



i i 



if 

\ 

uses were asked, Including subjective appraisals, personal imparts, open 

^ 4 li - 
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- other school or agency personnfl appreciated me toore 
, ' - pupils learned faster 



- It helped wlLh ^^-t^lio* 



"l.l ll, 
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centage " estlMte If necessary." (TollowoH by a list of spec- 
ific aJ.Cernativ^ ufees.) (Hnrne) 



"In yOui. Jud^ms-iiL. L^vc. t-^». r - t • i^t'*:^ n^KlL, a*i^rQ^ 



J-' 



. ,. « 1 f 1 , - 1 I » . , , . . ,i. ii 
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j;^^^ ^^Isr of the 

^llf^ J / u — V'^W^iiW rMoimfeBd wS^^fe^TOC^ the edu- ^ 

caiii uJefiil y^ti-^Be^ each 

^ = - imtBi '^n a S-^sist soala^^raoging^ frem^^^ery asefirij^. CQ "mQt at all 




IV* Reaeweh &tsigaj for ERIC Inpaet StudlM 



||||^' dlvMslCj^istfid chp^sc^ertsclci of the \farl^^|^ERI€ products and services 

^ ' prteludes aky slriiie research da^lgo to assBas thair iapact^ 

^^—^ ERIC has?^t0Cai-;COtttroJk^^^a^ <ha typa of produce^ a^d satvlces offered, 

their fdP^r/ irylei arrajigOTimt, afid fTequeney. " tey ^LaMeci change In one or" 
more of these cftajsct eristics fias the potential to be stud lad experimeatally 
such as by tctroduclng a product W.o^ly and it r^ndoaj ot by oaiQtainliig ^ . 
both Che old and new vera Ions with random disc rtbut Ion of each. For ^rapla, 
a new thes'^uiuaj^n be di^ttlbuted Co the SOCs In cvo wa-^es; Cha first wave tf 
a Ts^do^ half of the SOCa and chc second wava to the reiaatnlng SOCs, bu t ^im^ 



*1* * - - 

after coLlecctng iopact data IroiB aod-uaers or boch groups of SOCa. 
ittidiei can be ImpltBDeficad wheoe^&r rie^ organisation^ are add^^to the -ERIC 



Iq a slfflliar vmin^ experimental or ^ccftptablt ns.ii^'expexlaer 



_^=, syst^-as^ SOGs , EandomLy= pair tie.orgasiMtlQM and delay provJ 



oaterlala 



to 'Dtie set of the ^pairs until after che Impact studv is aoopietfd. 

Assasatng tht Impact of comptjter cspes^^^'-^e ri If^lcult be^iie^^^^P does 
oot pr^lde a computer aearchlng ^^rvjc^ Sofp« eprp^ritfiental 0^udlerf'<^f the^ 
cootpu^^J Capes may req"^ ^^'^■^ e^n p^rat ^^^^^^ t:^p?^^ # pfovtderf. " Ot , it 

may be possible tr^ produce tv/o verstt>^s ^ the c^-^ 'apes and distribute 



1 



r' 



ERIC 



^^ch v'ertloQ to a dlffereQt group of if^^lee providers. By j^^^^^ly 



Ill -36 



• Ing whleh BOCs to stud^^ m experiffleTitiJ. ox an acceptable no^liiperto&ntal 
dislipi Is .posMblet'^^ ^ i ^ ' . 

. ^ ^ ^ _ . ted JLastlyi 'uaei:^ md ^j^^tiaL users of carefully' selected SOCs sad CHs 



should be q^rled qe a regfili^ basis. Thim will begin M implfflSmC m regular 
study of jerformoce and tapact, ^ wdll obtain -mlcromeasur|s, and ic provides ' 
L"^he fleedeifl bafora aea^ure^^f any oneg of tlie*"£hree accapca^a not^^cpeilffleritai 
dBmigik are co ba carTled out at a l^er time* 




B, A Detailed Example, 



Information Amlysls Products are an tapbrtant mmd escpeasiye compb- 

neat of ERIC* \J^ti over 1000 ware prod^ffe^^^^l975. Several yearg. agq they _ 
occupted m larger portion of each CH*s budget (Wanger ^ reports 40-45t) ^ than they 
do now (abouc lQ-20%) , but chac is still 4 comsld^rablj 'dfoutit of EaSoey. Pre^ 



iumably the intetdad impact of the ffcs ii- tA maka the materials aore accessi- 




j and DO re understandable by^ and-uaeri* 



;s secclon will brleiLy riview WMger' study — tha only^t^dy whlc 



lAPs and will otitiina two rijearcft daslgng. wliich covld havtybaen- 
conducted whe^^Bie lAP^ were tn^iated, a^d to soma asctent,^ fiiti itiil be con-^^ 
djicted to asstss t hp * t**^^ impact , -s gsi^ ^ ^ 

m 



r 



I WagQer's study (ED 064 527-8)^ Wanger studlad tha reactloos\of a f*tde ^'t^^^ 
variety of educa^pTS a ^•""plp if rSIC CM products prap^rfd In ^SW^wf 

H^r flrpt ^li^sclonTiai^f ()'^ ^-^^^ ^ent to over 400Q. educator so-me of whom ^ 
ware fffelected rando^Jy ^n^<^ * resr rhosfn motk puVposafully froin E^IC malj 
Hstp anrf si^Xar anurrf.^*, tk^ rt^pona^ frot^^ Ql (return, rate 64%) was 
analyzed s^p-r^te <^h« f^xjca^.ors trira two categories: user^x^f and ^ ^ 



n 



ERIC 



[return ra€# ^ 



- V jQ. react* CO '10 swrQ^tEa of . E^C pToducts,^ Replies from q2 ( 
r \ . 6S^;-t»i[ete* divided Into a, t|'i^^^^d aon-raader category. The non-ueaifi ^fN^36L) 
= SCTt a third qaastionnalre (Q3)* fourth g^^tlooaaire was sMt to 



i^^^Jt^^saffiplS of eatici'tion ipeclallst 



IMC deeimeats tod react td^ th^ 



ERIC pvoiudtm wate aoaly^ad la tems ^^^^ " X-^. 

^ quality r coverage/ uprto^dat^neis, formapV clarify of witlng 

- ^ #\ ^ 

— tapae^i ;?a^ed only of* riadaM) , how wa4 used cliair 
^ everyday decisions ^ 



rhe major^lBding^ indlc 



ite that ERlC producti af^e knowp ^^posi^^^y 



^alued^ us ad and hmm^iL^^ m liMiXmd public, thfe '^rdlysis of ispaat,^ W^^iggr 

iouni that ERIC p^ducts assist educators in thatr everyday planii4n| 




abplicatloTis . research desig'Tt, -and ather prDfesslonal 4pclsibij=makii^ 
tlons - Readers of ERI& produces s^rve as lafqrffl^tlom sou^cas to othar educa-^, 
Cora ffiklclplyltil the Impact. And rfeadlfts fitelC bibLlogiaphles ife^^l^tHat 
they'^sa them Co find other helpful:.! ^fetfs ^pf linfor^^loa, Wan^jfjaoncludaa 
that the= products are favorably feceivaa^iit underuc ILte^d . She racommettd^d ^ 
that "t^o aeparate evalumtion stud las ba conducted In the futur^K^ 



An asiassnieric of khB outreajch of products, using both random 

X. . ^. ^ . ^ - . < ' ..... -.^-^ 

and nen-TandotiJ ^ triples, shouft ba mad^ per iod leal ly, partlcu^ 



larly after the i 



tn p 1 r?n e n r ^- 1 1 on 



iUn reman *t 



an Improved S 
' systOT or #f .ffiodlf IraCtf^ns m the d^ll\^erv system. 
A continuing eva luat ton-£f^o|raffl sho*-''^ be ns 1 1 tu ted to pro^ 
vide* the drlglr^a] Mnl^^ w^rh curr^nf reedh^ck on NCFf produet^.^ 



V. 



It Is very i ^r&tsr^in^ " 

%. 
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► nrtafi'^-e f-^n 'S'lchin the ^Dirlf 
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of EVOP. ^ttt tlia^'^kj co^OTifent of true experimentation is Mt mencloMd. This 
is an ii^ortanc coOTtderatloni "becauaa to matter how competently exacuted the 



^> study ^fHavalock notes^ tome TOataiesaes in the questionnaire methodology) , it^s 



uS^ &t"; Sff'dii :^aui t ie's^ar IsiSl from "s alB^s el mii ion o f r ispindeat s to-k^* 
*&:anElyais categories, tKSfe'fSs ueeta (both readers and nOT^rea4ets) andcnonr ^ »^ 



uaerSf 6f ^ICprgfiutfti may difftr on various pa 



rsbnat and ftrof eaaioital relateVi* .i^ jfti^ / 



vari^ffiles 



4jhiGh^will hao^fi any unmbiguous-^sesament 6f t^a Is^atit of s . 1, 
8, Or to give another exa^Ie, "EMC users may be reguJkf ^^^^^^S'^^ 



these producti 

seatcfters for new informtion* In such a *sltiiation, IC would not be justified? 
to conclude that'bne impact of ao^lRIC proceed bibliography Is ^ifeg^gumper^^^ 
xmm of iJev infomaclon Lt^m^^'^^m reader might have found thase IteMi-'br even 
"better Items, If the LAP ware aoC available/ Only a true ^l^d ^perifietxt^an 



cope with these potentlaj probliffls 



2, cyuj^f_ield e^erlment - In order to rmove, tt|e potentlal^roblem ol aelf^ 
selaetionl^her^t In survey reaearch* It wllj- be important to make a ^mpiffi^^ 

tlva analysis among group#of educator who a^e (or who, are no.t) uses^. 

' ' ' ' -"^^ ' ' ' - ' ^^^^ 

In this .ptrpposed. study ^e^ly ERIC users will be included- . ' / 

^ '^^fe- users need to^ Identified and randomly split d.ntp iiwo relgfiSLvel^ homo-t* 

geneous group%: those who rac^f^ed one or mota lA^ and thoai^ who hive not. 



Given the usets in^^Yn igroups r^iissted . the lAPs ^nd g%^^mn a la^ge enQiJgh 



sasgie Q^fedu^atori in each gro^^^thefr ^^^ diffarence betwgert the grQuiS^^ifa 



terms of various iopacs^^jpVsuV^ can bt atcribu^^ lA^ 



rhB study can ba conduce^- at a single CH or at ijiany CHs, And Mepatidlng 



on the acopa of the study, It ©ay^equlra tht cooperatlbn of EDRS'lfr orier 



to ^include' t^i%se users ^^ho ^ffqu^t^^X^i^ from ^ha& f^rce 



X h 



be U^gcrlbad heri ts^ In terms of/a slttgle^Cl 



r ^mpl^^y^ th 

We 



ERIC 




Wh^_^e^%sia for lAPs are^ rteiifcdi a coia^fllp la used to 
" Seeld wiil b€ honored* Ihose dei^ed 

t|i«^^tAP can' bs told JLt Is "otifr pi prtnC'-^e^he^^ cm be tsldn^^' 





^^'^^ai^Bm^ iir ttipBp dm±md ara toid about ^^s' studjr, eheee i*fao ware 



ndt denied should al§Q be told, 

date orf rtquest, eeqlaatort lAlvMqvsBted, 'and ^hechair -t^ re* 
^^^^^as fisn^edT^JCf the d^sand for eertain lAFs is parDd^ular-- 
ly heavy. It yould be aaeful to .§tratj,fy tha g rou^"^ In tjOTa of 



mimtpi^f denied Sad nQC^efiiffl re-- .*J£^& 



^ftPSi try to kftep an 
qaesia for , each of fiha^^^^W'^^ 
2. At some time later coneaet all requestors to determine the 




pact of tha^^IAPs.^ bibliographies one could ask tt the! cita- 

tions foiiii^ were prevlonsly unknown*' .Were thay^^read? Wert Che 
thought jtoj^be 
were they used 



usefu£2;.. ^?i^^he^^q.ta^ used? And if 
i? ^otm>^^^,{mfi^hm 'mmm questton^lll b^'aa^|^^E . 



those denied and tho^e no^f^nled. for revl^w^^rtielta %t 



ifiaasurmanc probleis are a "tilt sore difficult but oneflossibll^" 



ty is to interylew reg^j^grs of the same TJ^ and Msk what 
thiy have thought about and cofflpletad since tht^rdrq^asc. Were 
those dfinied &fel ilkely to*drop?^ a line of iftquifw Veraf those^ 

" ^ 1 ' 

j#?C ma«fer in the 1 



denied* pToductiva In terms ofjthe siibj#?c mat'ffer in the lAP? 

Did intake a longer cime ft? thfltep denied to diflGOver the In-- 
. formation ia the 1^7 ^^^^ ^ ^ 

Tha Irfl^frviews can be conduct id fcy mallj phogej oc ^ per% 
^ gpn, dependlril upon the tfepth an|H eomplexlCy o^f the speelf ±c 

• %. I 



ERIC ' 
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0C i 



qu^tiOT a^^^^.atid the budget of the itudy. It ±B impQt*^ 



tant^. however, co asGertain the fact that thosi"^denled am 
X4J by the CT*dld not obtada a copy elsewhere, ^ ~^ 



Thia"^tialag of thieVlttter^iefi/ ±s taportani and'^th^^^^ 
'Imi^e enough sMple size It would be beit to.'eonSuet the 
^tervlew over different *subsmta of requestors at different 
- . ^ ^taa pexipda- say two weeks %^4^rf^replyir^ 'to the request ^ 
plus oaef two, four,' and six iontlia Mfter, This wou^l per^ 
alt an aaaaeament of "cat^^--up" time^ whleh 4s aseottdftd 
%^0.tmti is in aooe seaee an i^act measure * 
3# jlfces^the study Ls co^leCed^ It may -be uaefuL to maintain a 



dec 




Non 



more limited profr^/Sf rahdoic^ demlals for lAPa a^roi^'-mll 
the CHa* If ps'^perl^ conducted can proyiBe a trust 
worthy fflechaplsitt to ^saess^iy / change in ^IC policy or 
proeednrea ^'^i.^t^i^S the lAP^, 



rilial I appro^ghas . Though it appeare that randomisation of end-^ 




*ls ' p^o ssiblfe yi r hlh a givfen CBt it may be too difficdl^06 carry out in 
;al pE^tlca/i', fi^htr researdi designs (II, HI,. IV) are possible. Hg^vf 



^^^t^efe axe sotaa^pecultar prdblCTs studying lAPg which makj^ these designs' less 



"Attractive altapnativ^es and more 




t Co implOTentu lAp\ are prodwats^ i 



of / CTs _and_^a_ are auM%gE-apptf±fi€ % Thergf ore^ if CTa are trtated aa dnltSj 



es between^he ^^^^^p 



trie subject i§At|.er oomfounds tl^^esu4ts'. That Ja, ^^11 the obtained differ- 



due to the ivailajbdtllty of the lAPs or due to ttie 
^tt«? Onfortuparely , there Is n6t any simple aolutlTO t# thfe ptqb^ 
lef^**^ a Sl^ue field CTp'tt^lsenc 1§ clearly tha bast cholqe* But assntolng that 
Ib l^osilbli, the rteict best alterriaci^es ire fo- ^ A 




ERIC • -'^^ 



limit thV conclu*. 



TOrk with only ome .Gi In ^si^ al^ aiid= Id 
slons to th^^d^usess of ctnit OH and to JChac subject mtk&c ^ 



umm pany CTs aoid mploj randoralMtlon to divide them^^tq 
^^l^^aps^W^an^ftSfrt^fy "equalise*- the ^groii^W "im feite^^Sf 7^ 



subject matter 




tepaet^ ij*ithe ^ttaata goal of a^ Infmraation Bjmtm. to Imv^'^aa ©f§Set on 
the'^liv^s^'^f its U4tfs< Howevar, studies ^of infomat ion system TSlPtily daal 
\* vith^Mpaet, primarily because it is difflEuit tp^TOsasu^e^ Enatsad, other 
g^ls or sub^goale (such as af f ect^venesa or efficiency) are analysEeil* "The 

recoisQend B^thods and ^e^sures^ which 
^euld be u#ed in future 'studi& of the liapict of ERIC*, Ge|^ainly few of these 



^b^^^ot fouSy^ their way int 



rJ^^mendations are niw* But because Che^« 
Teaaardh pTactice. t*hey b«4r repeat ing^. j- *\ = ^ 

' ■ i-^^ ...... 

ida'tlpD i^ tb use cruev^ifeid fexperfinents* 



^to^ The Mjor mathodblo^i 
Notts of the- existing '^EHlfe 
indings ttf be leas cl'^ftly* 




rii^eiitall";^ -^s^stng their 



pr|tahle. Gilh^jr^^ i^d-^tije colleagues re^^ewM 



resaareh studies in b^her human BmtftcB ar^s and concluded that^ 



"one characteristic of nosr^^^zed ^^ie& that^showj up 1^^'^^^ 
CO nj^iSL sfter cont^t is t^a t e van a ft « ^^^at^i^ i t ud i e s ha^ ^ . 
^^^nsistently^^^FtTOTtfie sape results, ^ere was always room <q 
^oubt Ihe validity of'^'f^ finding/* (p. 134)' C 



r 



^ > 



jrs fo^d It 



And, n/bte that chest. Wthors foiJnd lncerpr#tatton difficult even when several 
atu^ei sgr^#d . St^id|eai ERIC imp^^^ are too few a^tf'^l^o dlffier^ji to fo^ a 
rtltable b^sli^r eft^ljMt^nf fhe at^unt dT agra^4fit ^ong chelr reaui^s. • 



' 4r 



^ "'^J;m^^^^^^&Ltf^i^^m^AndBtion was mad^ Inateadj future rmaearcft-- 




irtl^ul^ii^^p belnji studied,, iast^^ Include tbe^ers' intmraetions 



wards tladlractniaage) , Frcm tthe llcwacure of social scierica fflaasurfflBent 
comas the suggestions to usa^-^^veral laeasures instead of onet t#feludiut the 
usa of unQbtruslva measures. ImprDve the ^liability and validity of all 
me^aur.as by pjetest^i;, repoving ambiguity s using critical incidents to., aid. 
training interviewer a^ eta* - /^""jf* 



recaM, ti 



/ Taken together, these suggestiona will Improve the state i^W^actlce of 
Jjnpact as^ssment * ^ 

i After surveving studies bf IHIC, Only *h could be foundi which contained 

*} ' ) 

f y^momm Impact component. Some of these studies were Impraseive scope and 1^ 
- execution (e.g., Asher L Vockell; Fry; Hoo^ Sieber; Wange 
studdes made a major or unique biraakthfDuJh in the ^easurCTei 




his Is not ^^rltlclsm but rat Her a tef lection of what 
^icture of the currenc state of lipact m'easurlSent 



Sigma to 'Bte.^ft adcu'^, 
Ne^ and p^^rf ul 

methods for measuring impact may not be easy tp come by, and the surest road « 



toward better impact rtudiaslls rhe systraatlc application o# piocedurei 



.J 



which have been CYIpH f^^'el|lted r^sp^fh fr^as 



r 
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^1 



ERIC 
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ehniques iSifeS* 



BouMe, C, ?• Review of the crl 
- evaltiatlOTi of rsf erene 
Confer enea on tfte Eval^ 




:ed for the 



:6£srenee Systems a^j Jracedurai * 



Bdx, G, di|i^Prafier^ N. R*. E^Qlutlonary?Operatlooi A Statistical^ Method £er 
iBBB Improv^gi^t / Nw Yotki Wiley, 1969. 



Cha^d 




arid DeLutla, T.^Q. ^multi-criterle approaeh to systCTs eval- 
on, Coffiputer and InforMtlon Science Res ear jcj^^ii^ater,^ Ohio 
te University, Colimbus, Ohio, May 1977* 



Clev^rdon» C, W, Design and evaluation of inforsiation syetOTia. Annual Rivjew 
1 . oi Infos^fflation Science and technology s 197=1^_.vqJ^ 6* =C,^/ Cuadra 
. (ed.), Chicago: Encyclopedia Brltannicai I«c. 

Cooks T, D, and Campbell, D. T. The design and conduct ^f quasi-e^perim^ts 
and true experiments in field settings* In H* Dunnette =(ed,), 
Handbook of Induatrial and Organizational Psychology , Chicago i Sand 
McKallyTl976, pp, "223-326. 



r, S. On selecting a measure of retrieval ef f ectiyeneis - I, Journal 
of ^he American Society for Infomatlon Science , March^Aprll^ 1973, 



& 



Cooper, W, On salectlng a measure of retrlaval effectiveness 

oj^th^e J^fflerican _Society for Inf prMtion Bttm^^% Nov^b 
1973, pp, 413-424. 




Debons, A, and^ontgomary , L, Design anrf ^eyaluatlon Iniormatidl^systMS . 

Annual Ravlew of I o rma c ion Sc i enc e^ gj^d^ T ee hno lo gy , 1974, vol. 9, 

C^ A* Cuadra (ed*)* Washington, 1^. C.i ^^erlcati Society for Informa'- 
-tion Science, A / 7 i 

Dervln, B., et a^l. The Development of Scratagies lot Dealing With the lnforma = 
tion Heeds of tJrban Residents ~ Phase I: The Citizen .S^udy. Flaal 
Report on Project . L0035JA, Grant Ho, QEG-O-W-7308 for the U^S, 
^Office of Education, Offtc# of Libraries and Learning gmouTCBB^ 1976. 
)-125-640.\ ■ 




and At^ertop* A, 
197#, 



ERIC Unm IdpntlfiRr Matrix. Mimeographed. Hay,% 




J. P,, Light, R, .J., and Hosteler, I' A^^^slng social thnovationf 
Ars ^trtpirical base <^o^iprMrv ^ a ^inn^r^ and ^ A ^^fcsd.^lo^ 



ERIC 





II1-45 



lipn and' &efiegjfaheat 



. fl^^orfefc ktgd&^i^mmB^^ 1975, j 



Series Expeyte^ltB J* Bpuideri GQiorad^'Assoaiated Ilj^^^alSy^^^ 
1975. ^ -^f ' ^ . . ^. = 



ock^ R, ERIC ay'aluafrlOB^^^^^ 



KattEi-^^ R, y/ Design evaluat^ps^^^^ffi^^^ftion* eystaai* ^mual Rav^ew oE 
' f ^ ^ Infrtmattoo Bblenee and TechTOlogjC 1969^ vsl* 4. Cuidra 

" GMomgoi * Smcjji^^^edls BrltaTOi^, pp* 31^70* 

Dxttel Llbrg^^iaar^gly , July 1977/ H^3), pp, ^S4«101, 

Edt^er, J.,*Cook, K* HC,- "and **Cr ouch, Wt Evaluatlflg Informatlgai A Guide tot 

Users of Social Science Resaar'fcti . Reading * f^ss * i Mdlson-^Wesley , 1-978. 

^ Kings D* Design and e^jaluatlon ^^infoTmation syatms, ^. .^^ualrRev^Lew tff In- 
formation SfilTOee and Tachnglo^ , 1968 , vol . 3 * C ,\A . Cuadra (ed * ) s 
Chicago: Efloy Wopedia Brltannica , pp ; 61-103. ' - i 





D, Atid BtjSD^ Tha evaluation of informaclon products and ser--/ ^ 

^ Vices* ^W^shig&oA's Iftformat Ion Resources Press, 1971. 

Laicaiter , F* . W. - Tha 'mfeasixremeoE^^ evaluation of library services . WaehingtQn 'a 
Information Resources Press, 1977* ^ - 



Laniahter., F, W. and Gillesplei C. J* Daslgn and eM^^atlon of Information sys- jf^ 
tarns. Arinual Review Qf_ Inf onylion Science aad_^TgchnolQ_gy , 1970, L ' 
^ vol^Bi A, jCuadra (ad,) , Chicago: Encyclo^dla TlriCtanniea- % -^^J 




jfi^ydoMt J* and Llff 



American PocfriB#ncatiqn, Octobar 1967 



^yeneral modal of J.nf omMtlt:^ -^ranWer* 



197-20; 



Iloal4.e--KeMann, Wantad: Rules for Wording S^rLcCured ^fc«cionxi^res , Public 
t^lnj^ qu arAr]^/ 1970^ 34, 191'2ai. 

Orr, R. D. Progress fe^tociuoencation ^ measurln^tha goodness of library service, 
Joucnal of Ph^entation , 1973, ^ (3)7?? ^ 31|-331. 




r 



i A> H# (The evaluaCl^ of r at rleval-- systems* ' Jras^ljnf OCT^t 
\Ser?^ , 1965, vol. 2, pp^ 129-146. • ' ' 



lon-Solanca 



^E/^Soiji DJ,<p^? I^ormEtlon ret*fc^val as a tf l^l-*and-error process ^i'^b^a^ 
Quiyt4rly /-^T»^7, y {2))^p- 128^1^- ^^»^ -^^'^^ ^ ' 




Swanaon, R. U-Y Performlhg/OTaluac^on studias Inf QrMtTOp?rtenct> Voiirn^of 
_ \^ ^ ^ th^^k^felqin Soc^ cy^for Iplora atiOD'gclaoce ^^ htey^June 197S,Vps 1?^^156. j 



aiuardon of informatTon sj^temij-. Aon^ ^ R|yley of ^ 
Infprma^lon ScLanc'e and^achnology , 1915, toI. JO. ' C , A ./tfioad|t s ( e d ' 



ERIC 



I I 



^';/Taylorp S. Queatlot-negdtlation aitd informatijon seeking^ m libraries. 
r " ^ : College atid Rasaarch Libraries , 1968, 29, pp, 178-194. _ ^ 

* Waias^ C. &rmluatlon Res ear eh, EnglOTood CUffs, N*J*r Prentice-Hall, 1972 



Zwei^lg, D. L* Meastitlng library uia, Drexel Library Quarterly , July 1977, 



Zwiezigi D* t* and Dervin, B. ^ Public library use, useri, uses ^ mdvm^m la 
% ^* . ' vtoa^d,ad|a of eharacterlstlcs and needs of. the adult clleatale ^f ^ 
toaticari public libraries. Advaflces in Llbrarlanshlp -T , 4977y = 
pp. 231^255. "~ " . 






\ 



FRIC 



r i 



